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Example - Asteroid 433 Eros (NEAR mission) 

• N optical images; very large number N> 10000 

• Light direction ~ known 

• Camera parameters ~ known 

• Albedo ~ constant 



N calibrated images 


Geometry 
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N aerial or satellite images 


Cameras 


Light sources 


Elevation (height field) 


Case 2 
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infer topography and scalar albedo 


Wbeclo (panchromatic) 
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3D reconstruction: existing methods I 




Shape from Stereo [Zhang et ai. 94] 



The 2D projection of a surface point into 
the left and right cameras can be re- 
projected to give a point in the 3D space 


Drawbacks : 

• Relies on finding point matches in both 
imgages 

• The density of the recovered surface 
points is data dependent 


Shape from Shading [Hom & Brooks 89] 

• Image gradients are related directly to 
surface gradients. 

• Integrabng from known boundary 
condition gives heights. 



Drawbacks : 

• Density of points is fixed^at Jhe image 
resolution 

• Assumptions: Lambertian reflection 
and constant albedo 


6/45 




3D reconstruction: existing methods II 
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Generalized stereo [Fua & Lederc 94] 



Use textured triangles on a dense 3D 
mesh as stereo features 
Optimize a cost function including a data 
term and a smoothness penalty term 
Combines stereo and shading information 


Drawbacks : 

• Not a Bayesian method: difficult to infer 
prior model parameters, and the data 
term is not natural 


Other methods 


[Morris & Kanade 03] 

Keep vertices fixed, search for good 
triangulations 

[Isidoro & Sclaroff 02, Vogiatzis et al. 03] 

Stochastic algorithm 

Iteratively shrink and deform a mesh 


Drawbacks : 

• Stochastic: slow, complex 

• Others: sensitive to local optima 
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Advantages of the proposed approach 


Ames Research Center 



Existing methods are not compatible with our long-term goal: 


. ~ - 

dense reconstruction of both geometry and reflectance, 

taking advantage of all the information in the N images 


The proposed approach : 

- Enables us to arbitrarily choose the surface density 
(e.g. according to the amount of available data 

- Can use any reflectance model, 

from the simple Lambertian BRDF to complex realistic BRDFs 

- Can use data from any sensor, whatever the pixel size; 

in principle we should be able to use non-optical data (e.g. altimetry) 

- Should be able to easily integrate new images 
once a initial estimate has been computed 
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The Bayesian approach 


posterior density 

5 {0} m 

^(surface, cameras I images) cc 

^(surface) x camera,) x J~J ^surface., camera,) 

V ' ' ^ ' 

prior density likelihood 


Surface 

Prior 


model 

camera pose 

model !; 

Gaussian fractal model 

: Gauss/arr model 
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Graphical model (generative model) 



• Observation parameter 

- Mean: calibration 
-.Uncertainty on. camera pose 


Graphical model : 

Relationship between random variables 
(prior and conditional densities) 
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The different densities 
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S J Surface model P(S) 

Geometry: dense triangular mesh: vertices + neighborhood 
Reflectance: one albedo per vertex 
Topology: arbitrary - DEM (flat), planet/asteroid (spherical) 
Subdivided mesh, has the topology of the initial mesh 


o 


Observation parameter density P(0) 

Camera pose: position, orientatio 

Camera physics : PSF, noise variance 

Light source: orientation, ambient and direct intensity 

Image formation model P(Y / 1(5,0) ) 

Rendering: realistic synthetic image, I(S,0) non-linear w.r.t. S,0 
Degradation: blur by PSF, Gaussian noise 
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Surface analysis and modeling 


What tool to use to analyze and model fractal surfaces? 

Check the statistical self-similarity: 

first compute the power spectrum of the object to analyze. 

• Height fields on planar surfaces: 2D Fourier Transform 

• Height fields on spherical surfaces: Spherical Harmonics 

• Arbitrary surfaces : ? 

irregular sampling, arbitrary topology 


Wavelets on subdivided meshes 

Spectrum: feature size = fct(scale) 
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Topology, geometry and regularity 


Topological support 

Set of sites (vertices) + neighborhood system 
Support regularity = neighborhood regularity 

Semi-regular mesh: 5 or 6 neighbors 


Geometry £ 

3D point for each topological site §, _ 

Objects can have an irregular geometry, 
but the wavelets are defined on a semi- regular 
topological support. 


topology 


topology 











Wavelet coefficients at level j+l = 
vertices at level j+l - prediction from level j 


encode the details at level j+l 


Vertices 


prediction 




approximation 


details 
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Multiresolution analysis 
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A new multiscale model 
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Scale-invariant adaptive Gaussian model on 

prior roughness of the surface 


P ({**■' i}) oc TT exp r — A» (Si) 2? k'i±| 2 ) 



geometric details = 
Gaussian random variables 



local scales 


spatially adaptive 
parameters 


■ 3D analog of the fractional Brownian motion in 2D 

(wavelet coefficients instead of Fourier coefficients) 

Efficient description of the power spectrum of natural images 


• Statistical model of wi-i- : mesh regularity prior density 

(sampling regularity) 
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Observation parameters 




prior camera pose : mean 



uncertainty : covariance 2 


Prior on the observation parameters (camera, light, PSF, noise) : 

• Knowledge of the camera motion and orientation 
(rover: odometers; space probe: gyroscopes, etc.) 

• Use the navigation parameters if available 

• Sensors: orientation w.r.t. sun or stars 

• Prior knowledge of the PSF and noise level 
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Forward problem: rendering 


/ * W- 

f * 

Image 

7 ( 5 . 0 ) 


Rendering : 

• known surface 

• known reflectance J 

• known light source 

• known camera param. 
(internal & external) 




Surface S 


Compute the intensity 

for each pixel 

... and its derivatives ! 


j/~ \ Why do we need an accurate rendering? 
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Sub-pixel accuracy required for good reconstruction accuracy and 
super-resolution 

Compute the derivatives of the pixel intensity w.r.t. all the 
parameters (surface, camera, light) to perform optimization 
Photometric accuracy: ambient light, reflectance functions, etc. 
Occlusions (hidden surface removal) 

Shadows (remove surfaces hidden from the light source) 


We need to work in the object space 


Most of existing algorithms work in image space 
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Image space or object space? 


Image 



Chief 



' Very fast, real time 
(OpenGL, etc.) 

1 Image = non-continuous 
function of the surface S 
Aliasing 

■ Limited to big triangles 


• Sum of contributions / pixel 

I, 

A 

• Image = continuous fct. of S 

• No aliasing (better sampling) 

• Works with any triangle size 

• Independence image/model 
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The rendering method 


Core of the algorithm : occlusion removal 

Hidden surfaces = occlusions / camera projection 
Surfaces in shadow = occlusions / light source projection 


Compute the contribution to a pixel 
by a partially hidden triangle 

i Po • Polygonal approach 

* • Recursive polygon-triangle subtractions 

• Trick: ^ 4 

only on ridge lines _ 


At the end: polygon moments + derivatives 
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The rendering tree. 




jr\r: under 

! -.“.A came ^|j development 



irradiances 
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Derivative computation 
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Examples of derivatives 


The chain rule 


parallel 


projected vertex 


normal 


2D polygon vertex 


irradiance 


polygon area 


pixel intensity 


Rendered images: asteroid 
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Rendering examples 1 



Light source rotation 
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Rendering examples 2 


Object rotation 
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• At the end : keep the inverse covariance matrix (related to the 
uncertainty), not only the estimated parameters S, {0} 


• Very useful to initialize a new estimation procedure, 
so that new data can be added 

• recursive refinement of S, {©}: 

- Update the 3D (process large amounts of data recursively instead of batch) 

- Refine the camera pose - Simultaneous Localization And Mappping (SLAM) 


36/45 


18 




Ames Research Center 


Preliminary surface reconstruction results 

Duckwater, Nevada 



• Physical model built using USGS elevation model 


• Hand-painted albedo 

Images : CMOS camera 

(10 bits, monochromatic) 
Light source : sun 


Goal: reconstruct DEM+albedo 
Assumptions: 

no shadows, no occlusions [old] 
Preliminary estimation : 

• sun direction (sun dial) 

• camera param. (checkerboard) 

One of the 8 observed images 

37/45 




A 

' Amps Rp 


Preliminary surface reconstruction results 

Ames Research Center DuCKWater, Nevada 


300 400 900 900 -TOO — 300 —4m 9 «0 900 TOO 

Ground truth (USGS DEM) Inferred DEM 

(max error<10mm, RMS error<2mm ) 

38/45 


Geometry (height field) 
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Potential applications 




• 3D object reconstruction from multiple images: 

- Asteroids (albedo ^uniform, spherical topology) 

- Planetary surfaces (variable albedo, planar/spherical topology) 

- Rover situation: MER mission (variable albedo, overhangs) 

• Simultaneous Localization And Mapping (SLAM): 

- Simplified terrain model: use 3D features instead of meshes 


• Space probe localization and 3D object recovery: 

- approach and flyby: recursive trajectory estimation and ol 
model refinement 

• Multi-sensor data fusion: 

- Optical (multi- and hyperspectral), radar (SAR), lidar, 

• Fractal geometry and synthetic images: 

- Generate realistic fractal surfaces 

- Compute photorealistic BRDFs for natural surfaces 
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Inverse rendering and computer vision 
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Computer vision 

(3D model reconstruction from multiple 
observations): 

inverse problem of rendering 

Bayesian inference 

applied to this inverse problem: 
everything is described by random variables 

2D data fusion into a single 3D model 
becomes a parameter estimation problem 


• It can be solved by existing efficient 

optimization techniques 
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Main contributions 
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• Rendering: 

- Take into account shadows and occlusions 

- Visibility polygons (recursive subtraction) 

- Derivative computation 

- Extensions 



• Surface modeling: 

~new wavelet transform on surfaces of 
arbitrary topology: 

- normal/parallel wavelet detail separation 

- local scale computation 

- Fractal model for natural surfaces 
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Extensions and future work I - Rendering 
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• Extensions (in progress): 

- Accurate shadows 

- Continuous shading 
(visibility polygon moments) 

- Continuous PSF, 

take into account the blur 

- complex BRDFs (* Lambert) 



• Future extensions: 

- Multispectral 

- Adaptive subdivision 

- Adaptive BRDFs 

- Secondary reflections 

- Other types of camera (push-broom, etc.) 

- Approximate methods 
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Extensions and future work II 
models and inference 




• Model extensions: 

Study real surfaces (Earth, Mars): 

- Reflectance/geometry interactions 

- Multi- and hyperspectral albedos 



• Inference method extensions: 

Marginalization: 

- Simultaneous reconstruction and calibration 

- Separated geometry and albedo inference 
Bayesian model selection: 

- Infer the topology (overhangs, etc.) 

- Dynamic and adaptive subdivision 
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